{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Neural network hybrid recommendation system on Google Analytics data model and training\n",
    "\n",
    "This notebook demonstrates how to implement a hybrid recommendation system using a neural network to combine content-based and collaborative filtering recommendation models using Google Analytics data. We are going to use the learned user embeddings from [wals.ipynb](../wals.ipynb) and combine that with our previous content-based features from [content_based_using_neural_networks.ipynb](../content_based_using_neural_networks.ipynb)\n",
    "\n",
    "Now that we have our data preprocessed from BigQuery and Cloud Dataflow, we can build our neural network hybrid recommendation model to our preprocessed data. Then we can train locally to make sure everything works and then use the power of Google Cloud ML Engine to scale it out."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're going to use TensorFlow Hub to use trained text embeddings, so let's first pip install that and reset our session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
     "!pip3 install tensorflow_hub"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
     "%%bash\n",
     "pip install --upgrade tensorflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now reset the notebook's session kernel! Since we're no longer using Cloud Dataflow, we'll be using the python3 kernel from here on out so don't forget to change the kernel if it's still python2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import helpful libraries and setup our project, bucket, and region\n",
    "import os\n",
    "import tensorflow as tf\n",
    "import tensorflow_hub as hub\n",
    "\n",
    "# PROJECT = \"cloud-training-demos\" # REPLACE WITH YOUR PROJECT ID\n",
    "# BUCKET = \"cloud-training-demos-ml\" # REPLACE WITH YOUR BUCKET NAME\n",
    "# REGION = \"us-central1\" # REPLACE WITH YOUR BUCKET REGION e.g. us-central1\n",
    "PROJECT = \"qwiklabs-gcp-cbc8684b07fc2dbd\" # REPLACE WITH YOUR PROJECT ID\n",
    "BUCKET = \"qwiklabs-gcp-cbc8684b07fc2dbd-bucket\" # REPLACE WITH YOUR BUCKET NAME\n",
    "REGION = \"us-east1\" # REPLACE WITH YOUR BUCKET REGION e.g. us-central1\n",
    "\n",
    "# do not change these\n",
    "os.environ[\"PROJECT\"] = PROJECT\n",
    "os.environ[\"BUCKET\"] = BUCKET\n",
    "os.environ[\"REGION\"] = REGION\n",
    "os.environ[\"TFVERSION\"] = \"1.13\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Updated property [core/project].\n",
      "Updated property [compute/region].\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "gcloud config set project $PROJECT\n",
    "gcloud config set compute/region $REGION"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "if ! gcloud storage ls | grep -q gs://${BUCKET}/hybrid_recommendation/preproc; then\n",    "    gcloud storage buckets create --location=${REGION} gs://${BUCKET}\n",    "    # copy canonical set of preprocessed files if you didn't do preprocessing notebook\n",
    "    gcloud storage cp --recursive gs://cloud-training-demos/courses/machine_learning/deepdive/10_recommendation/hybrid_recommendation gs://${BUCKET}\n",    "fi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2> Create hybrid recommendation system model using TensorFlow </h2>\n",
    "\n",
    "Now that we've created our training and evaluation input files as well as our categorical feature vocabulary files, we can create our TensorFlow hybrid recommendation system model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's first get some of our aggregate information that we will use in the model from some of our preprocessed files we saved in Google Cloud Storage."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.python.lib.io import file_io"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number_of_content_ids = 15634\n"
     ]
    }
   ],
   "source": [
    "# Get number of content ids from text file in Google Cloud Storage\n",
    "with file_io.FileIO(tf.gfile.Glob(filename = \"gs://{}/hybrid_recommendation/preproc/vocab_counts/content_id_vocab_count.txt*\".format(BUCKET))[0], mode = 'r') as ifp:\n",
    "    number_of_content_ids = int([x for x in ifp][0])\n",
    "print(\"number_of_content_ids = {}\".format(number_of_content_ids))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number_of_categories = 3\n"
     ]
    }
   ],
   "source": [
    "# Get number of categories from text file in Google Cloud Storage\n",
    "with file_io.FileIO(tf.gfile.Glob(filename = \"gs://{}/hybrid_recommendation/preproc/vocab_counts/category_vocab_count.txt*\".format(BUCKET))[0], mode = 'r') as ifp:\n",
    "    number_of_categories = int([x for x in ifp][0])\n",
    "print(\"number_of_categories = {}\".format(number_of_categories))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number_of_authors = 1103\n"
     ]
    }
   ],
   "source": [
    "# Get number of authors from text file in Google Cloud Storage\n",
    "with file_io.FileIO(tf.gfile.Glob(filename = \"gs://{}/hybrid_recommendation/preproc/vocab_counts/author_vocab_count.txt*\".format(BUCKET))[0], mode = 'r') as ifp:\n",
    "    number_of_authors = int([x for x in ifp][0])\n",
    "print(\"number_of_authors = {}\".format(number_of_authors))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mean_months_since_epoch = 573.60733908\n"
     ]
    }
   ],
   "source": [
    "# Get mean months since epoch from text file in Google Cloud Storage\n",
    "with file_io.FileIO(tf.gfile.Glob(filename = \"gs://{}/hybrid_recommendation/preproc/vocab_counts/months_since_epoch_mean.txt*\".format(BUCKET))[0], mode = 'r') as ifp:\n",
    "    mean_months_since_epoch = float([x for x in ifp][0])\n",
    "print(\"mean_months_since_epoch = {}\".format(mean_months_since_epoch))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Determine CSV and label columns\n",
    "NON_FACTOR_COLUMNS = \"next_content_id,visitor_id,content_id,category,title,author,months_since_epoch\".split(',')\n",
    "FACTOR_COLUMNS = [\"user_factor_{}\".format(i) for i in range(10)] + [\"item_factor_{}\".format(i) for i in range(10)]\n",
    "CSV_COLUMNS = NON_FACTOR_COLUMNS + FACTOR_COLUMNS\n",
    "LABEL_COLUMN = \"next_content_id\"\n",
    "\n",
    "# Set default values for each CSV column\n",
    "NON_FACTOR_DEFAULTS = [[\"Unknown\"],[\"Unknown\"],[\"Unknown\"],[\"Unknown\"],[\"Unknown\"],[\"Unknown\"],[mean_months_since_epoch]]\n",
    "FACTOR_DEFAULTS = [[0.0] for i in range(10)] + [[0.0] for i in range(10)] # user and item\n",
    "DEFAULTS = NON_FACTOR_DEFAULTS + FACTOR_DEFAULTS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create input function for training and evaluation to read from our preprocessed CSV files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create input function for train and eval\n",
    "def read_dataset(filename, mode, batch_size = 512):\n",
    "    def _input_fn():\n",
    "        def decode_csv(value_column):\n",
    "            columns = tf.decode_csv(records = value_column, record_defaults = DEFAULTS)\n",
    "            features = dict(zip(CSV_COLUMNS, columns))          \n",
    "            label = features.pop(LABEL_COLUMN)         \n",
    "            return features, label\n",
    "\n",
    "        # Create list of files that match pattern\n",
    "        file_list = tf.gfile.Glob(filename = filename)\n",
    "\n",
    "        # Create dataset from file list\n",
    "        dataset = tf.data.TextLineDataset(filenames = file_list).map(map_func = decode_csv)\n",
    "\n",
    "        if mode == tf.estimator.ModeKeys.TRAIN:\n",
    "            num_epochs = None # indefinitely\n",
    "            dataset = dataset.shuffle(buffer_size = 10 * batch_size)\n",
    "        else:\n",
    "            num_epochs = 1 # end-of-input after this\n",
    "\n",
    "        dataset = dataset.repeat(count = num_epochs).batch(batch_size = batch_size)\n",
    "        return dataset.make_one_shot_iterator().get_next()\n",
    "    return _input_fn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will create our feature columns using our read in features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create feature columns to be used in model\n",
    "def create_feature_columns(args):\n",
    "    # Create content_id feature column\n",
    "    content_id_column = tf.feature_column.categorical_column_with_hash_bucket(\n",
    "        key = \"content_id\",\n",
    "        hash_bucket_size = number_of_content_ids)\n",
    "\n",
    "    # Embed content id into a lower dimensional representation\n",
    "    embedded_content_column = tf.feature_column.embedding_column(\n",
    "        categorical_column = content_id_column,\n",
    "        dimension = args[\"content_id_embedding_dimensions\"])\n",
    "\n",
    "    # Create category feature column\n",
    "    categorical_category_column = tf.feature_column.categorical_column_with_vocabulary_file(\n",
    "        key = \"category\",\n",
    "        vocabulary_file = tf.gfile.Glob(filename = \"gs://{}/hybrid_recommendation/preproc/vocabs/category_vocab.txt*\".format(args[\"bucket\"]))[0],\n",
    "        num_oov_buckets = 1)\n",
    "\n",
    "    # Convert categorical category column into indicator column so that it can be used in a DNN\n",
    "    indicator_category_column = tf.feature_column.indicator_column(categorical_column = categorical_category_column)\n",
    "\n",
    "    # Create title feature column using TF Hub\n",
    "    embedded_title_column = hub.text_embedding_column(\n",
    "        key = \"title\", \n",
    "        module_spec = \"https://tfhub.dev/google/nnlm-de-dim50-with-normalization/1\",\n",
    "        trainable = False)\n",
    "\n",
    "    # Create author feature column\n",
    "    author_column = tf.feature_column.categorical_column_with_hash_bucket(\n",
    "        key = \"author\",\n",
    "        hash_bucket_size = number_of_authors + 1)\n",
    "\n",
    "    # Embed author into a lower dimensional representation\n",
    "    embedded_author_column = tf.feature_column.embedding_column(\n",
    "        categorical_column = author_column,\n",
    "        dimension = args[\"author_embedding_dimensions\"])\n",
    "\n",
    "    # Create months since epoch boundaries list for our binning\n",
    "    months_since_epoch_boundaries = list(range(400, 700, 20))\n",
    "\n",
    "    # Create months_since_epoch feature column using raw data\n",
    "    months_since_epoch_column = tf.feature_column.numeric_column(\n",
    "        key = \"months_since_epoch\")\n",
    "\n",
    "    # Create bucketized months_since_epoch feature column using our boundaries\n",
    "    months_since_epoch_bucketized = tf.feature_column.bucketized_column(\n",
    "        source_column = months_since_epoch_column,\n",
    "        boundaries = months_since_epoch_boundaries)\n",
    "\n",
    "    # Cross our categorical category column and bucketized months since epoch column\n",
    "    crossed_months_since_category_column = tf.feature_column.crossed_column(\n",
    "        keys = [categorical_category_column, months_since_epoch_bucketized],\n",
    "        hash_bucket_size = len(months_since_epoch_boundaries) * (number_of_categories + 1))\n",
    "\n",
    "    # Convert crossed categorical category and bucketized months since epoch column into indicator column so that it can be used in a DNN\n",
    "    indicator_crossed_months_since_category_column = tf.feature_column.indicator_column(\n",
    "            categorical_column = crossed_months_since_category_column)\n",
    "\n",
    "    # Create user and item factor feature columns from our trained WALS model\n",
    "    user_factors = [tf.feature_column.numeric_column(key = \"user_factor_\" + str(i)) for i in range(10)]\n",
    "    item_factors =  [tf.feature_column.numeric_column(key = \"item_factor_\" + str(i)) for i in range(10)]\n",
    "\n",
    "    # Create list of feature columns\n",
    "    feature_columns = [embedded_content_column,\n",
    "    embedded_author_column,\n",
    "    indicator_category_column,\n",
    "    embedded_title_column,\n",
    "    indicator_crossed_months_since_category_column] + user_factors + item_factors\n",
    "\n",
    "    return feature_columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll create our model function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create custom model function for our custom estimator\n",
    "def model_fn(features, labels, mode, params):\n",
    "    # Create neural network input layer using our feature columns defined above\n",
    "    net = tf.feature_column.input_layer(features = features, feature_columns = params[\"feature_columns\"])\n",
    "\n",
    "    # Create hidden layers by looping through hidden unit list\n",
    "    for units in params[\"hidden_units\"]:\n",
    "        net = tf.layers.dense(inputs = net, units = units, activation = tf.nn.relu)\n",
    "\n",
    "    # Compute logits (1 per class) using the output of our last hidden layer\n",
    "    logits = tf.layers.dense(inputs = net, units = params[\"n_classes\"], activation = None)\n",
    "\n",
    "    # Find the predicted class indices based on the highest logit (which will result in the highest probability)\n",
    "    predicted_classes = tf.argmax(input = logits, axis = 1)\n",
    "\n",
    "  # Read in the content id vocabulary so we can tie the predicted class indices to their respective content ids\n",
    "    with file_io.FileIO(tf.gfile.Glob(filename = \"gs://{}/hybrid_recommendation/preproc/vocabs/content_id_vocab.txt*\".format(BUCKET))[0], mode = \"r\") as ifp:\n",
    "        content_id_names = tf.constant(value = [x.rstrip() for x in ifp])\n",
    "\n",
    "    # Gather predicted class names based predicted class indices\n",
    "    predicted_class_names = tf.gather(params = content_id_names, indices = predicted_classes)\n",
    "\n",
    "    # If the mode is prediction\n",
    "    if mode == tf.estimator.ModeKeys.PREDICT:\n",
    "        # Create predictions dict\n",
    "        predictions_dict = {\n",
    "            \"class_ids\": tf.expand_dims(input = predicted_classes, axis = -1),\n",
    "            \"class_names\" : tf.expand_dims(input = predicted_class_names, axis = -1),\n",
    "            \"probabilities\": tf.nn.softmax(logits = logits),\n",
    "            \"logits\": logits\n",
    "        }\n",
    "\n",
    "        # Create export outputs\n",
    "        export_outputs = {\"predict_export_outputs\": tf.estimator.export.PredictOutput(outputs = predictions_dict)}\n",
    "\n",
    "        return tf.estimator.EstimatorSpec( # return early since we\"re done with what we need for prediction mode\n",
    "          mode = mode,\n",
    "          predictions = predictions_dict,\n",
    "          loss = None,\n",
    "          train_op = None,\n",
    "          eval_metric_ops = None,\n",
    "          export_outputs = export_outputs)\n",
    "\n",
    "    # Continue on with training and evaluation modes\n",
    "\n",
    "    # Create lookup table using our content id vocabulary\n",
    "    table = tf.contrib.lookup.index_table_from_file(\n",
    "        vocabulary_file = tf.gfile.Glob(filename = \"gs://{}/hybrid_recommendation/preproc/vocabs/content_id_vocab.txt*\".format(BUCKET))[0])\n",
    "\n",
    "    # Look up labels from vocabulary table\n",
    "    labels = table.lookup(keys = labels)\n",
    "\n",
    "    # Compute loss using sparse softmax cross entropy since this is classification and our labels (content id indices) and probabilities are mutually exclusive\n",
    "    loss = tf.losses.sparse_softmax_cross_entropy(labels = labels, logits = logits)\n",
    "\n",
    "    # If the mode is evaluation\n",
    "    if mode == tf.estimator.ModeKeys.EVAL:\n",
    "        # Compute evaluation metrics of total accuracy and the accuracy of the top k classes\n",
    "        accuracy = tf.metrics.accuracy(labels = labels, predictions = predicted_classes, name = \"acc_op\")\n",
    "        top_k_accuracy = tf.metrics.mean(values = tf.nn.in_top_k(predictions = logits, targets = labels, k = params[\"top_k\"]))\n",
    "        map_at_k = tf.metrics.average_precision_at_k(labels = labels, predictions = predicted_classes, k = params[\"top_k\"])\n",
    "\n",
    "        # Put eval metrics into a dictionary\n",
    "        eval_metric_ops = {\n",
    "            \"accuracy\": accuracy,\n",
    "            \"top_k_accuracy\": top_k_accuracy,\n",
    "            \"map_at_k\": map_at_k}\n",
    "\n",
    "        # Create scalar summaries to see in TensorBoard\n",
    "        tf.summary.scalar(name = \"accuracy\", tensor = accuracy[1])\n",
    "        tf.summary.scalar(name = \"top_k_accuracy\", tensor = top_k_accuracy[1])\n",
    "        tf.summary.scalar(name = \"map_at_k\", tensor = map_at_k[1])\n",
    "        \n",
    "        return tf.estimator.EstimatorSpec( # return early since we\"re done with what we need for evaluation mode\n",
    "            mode = mode,\n",
    "            predictions = None,\n",
    "            loss = loss,\n",
    "            train_op = None,\n",
    "            eval_metric_ops = eval_metric_ops,\n",
    "            export_outputs = None)\n",
    "\n",
    "    # Continue on with training mode\n",
    "\n",
    "    # If the mode is training\n",
    "    assert mode == tf.estimator.ModeKeys.TRAIN\n",
    "\n",
    "    # Create a custom optimizer\n",
    "    optimizer = tf.train.AdagradOptimizer(learning_rate = params[\"learning_rate\"])\n",
    "\n",
    "    # Create train op\n",
    "    train_op = optimizer.minimize(loss = loss, global_step = tf.train.get_global_step())\n",
    "\n",
    "    return tf.estimator.EstimatorSpec( # final return since we\"re done with what we need for training mode\n",
    "        mode = mode,\n",
    "        predictions = None,\n",
    "        loss = loss,\n",
    "        train_op = train_op,\n",
    "        eval_metric_ops = None,\n",
    "        export_outputs = None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now create a serving input function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create serving input function\n",
    "def serving_input_fn():  \n",
    "    feature_placeholders = {\n",
    "        colname : tf.placeholder(dtype = tf.string, shape = [None]) \\\n",
    "        for colname in NON_FACTOR_COLUMNS[1:-1]\n",
    "    }\n",
    "    feature_placeholders[\"months_since_epoch\"] = tf.placeholder(dtype = tf.float32, shape = [None])\n",
    "\n",
    "    for colname in FACTOR_COLUMNS:\n",
    "        feature_placeholders[colname] = tf.placeholder(dtype = tf.float32, shape = [None])\n",
    "\n",
    "    features = {\n",
    "        key: tf.expand_dims(tensor, -1) \\\n",
    "        for key, tensor in feature_placeholders.items()\n",
    "    }\n",
    "\n",
    "    return tf.estimator.export.ServingInputReceiver(features = features, receiver_tensors = feature_placeholders)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that all of the pieces are assembled let's create and run our train and evaluate loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create train and evaluate loop to combine all of the pieces together.\n",
    "tf.logging.set_verbosity(tf.logging.INFO)\n",
    "def train_and_evaluate(args):\n",
    "    estimator = tf.estimator.Estimator(\n",
    "        model_fn = model_fn,\n",
    "        model_dir = args[\"output_dir\"],\n",
    "        params = {\n",
    "        \"feature_columns\": create_feature_columns(args),\n",
    "        \"hidden_units\": args[\"hidden_units\"],\n",
    "        \"n_classes\": number_of_content_ids,\n",
    "        \"learning_rate\": args[\"learning_rate\"],\n",
    "        \"top_k\": args[\"top_k\"],\n",
    "        \"bucket\": args[\"bucket\"]\n",
    "        }\n",
    "    )\n",
    "\n",
    "    train_spec = tf.estimator.TrainSpec(\n",
    "        input_fn = read_dataset(filename = args[\"train_data_paths\"], mode = tf.estimator.ModeKeys.TRAIN, batch_size = args[\"batch_size\"]),\n",
    "        max_steps = args[\"train_steps\"])\n",
    "\n",
    "    exporter = tf.estimator.LatestExporter(name = \"exporter\", serving_input_receiver_fn = serving_input_fn)\n",
    "\n",
    "    eval_spec = tf.estimator.EvalSpec(\n",
    "        input_fn = read_dataset(filename = args[\"eval_data_paths\"], mode = tf.estimator.ModeKeys.EVAL, batch_size = args[\"batch_size\"]),\n",
    "        steps = None,\n",
    "        start_delay_secs = args[\"start_delay_secs\"],\n",
    "        throttle_secs = args[\"throttle_secs\"],\n",
    "        exporters = exporter)\n",
    "\n",
    "    tf.estimator.train_and_evaluate(estimator = estimator, train_spec = train_spec, eval_spec = eval_spec)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run train_and_evaluate!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:vocabulary_size = 3 in category is inferred from the number of elements in the vocabulary_file gs://qwiklabs-gcp-cbc8684b07fc2dbd-bucket/hybrid_recommendation/preproc/vocabs/category_vocab.txt-00000-of-00001.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:29:45.557669 139814528628480 feature_column_v2.py:1625] vocabulary_size = 3 in category is inferred from the number of elements in the vocabulary_file gs://qwiklabs-gcp-cbc8684b07fc2dbd-bucket/hybrid_recommendation/preproc/vocabs/category_vocab.txt-00000-of-00001.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using default config.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:30:01.019747 139814528628480 estimator.py:1739] Using default config.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using config: {'_save_checkpoints_secs': 600, '_device_fn': None, '_log_step_count_steps': 100, '_task_type': 'worker', '_evaluation_master': '', '_experimental_distribute': None, '_task_id': 0, '_session_config': allow_soft_placement: true\n",
      "graph_options {\n",
      "  rewrite_options {\n",
      "    meta_optimizer_iterations: ONE\n",
      "  }\n",
      "}\n",
      ", '_global_id_in_cluster': 0, '_save_summary_steps': 100, '_master': '', '_is_chief': True, '_train_distribute': None, '_tf_random_seed': None, '_save_checkpoints_steps': None, '_model_dir': 'hybrid_recommendation_trained', '_keep_checkpoint_max': 5, '_service': None, '_num_ps_replicas': 0, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f29180a88d0>, '_keep_checkpoint_every_n_hours': 10000, '_protocol': None, '_eval_distribute': None, '_num_worker_replicas': 1}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:30:01.021662 139814528628480 estimator.py:201] Using config: {'_save_checkpoints_secs': 600, '_device_fn': None, '_log_step_count_steps': 100, '_task_type': 'worker', '_evaluation_master': '', '_experimental_distribute': None, '_task_id': 0, '_session_config': allow_soft_placement: true\n",
      "graph_options {\n",
      "  rewrite_options {\n",
      "    meta_optimizer_iterations: ONE\n",
      "  }\n",
      "}\n",
      ", '_global_id_in_cluster': 0, '_save_summary_steps': 100, '_master': '', '_is_chief': True, '_train_distribute': None, '_tf_random_seed': None, '_save_checkpoints_steps': None, '_model_dir': 'hybrid_recommendation_trained', '_keep_checkpoint_max': 5, '_service': None, '_num_ps_replicas': 0, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f29180a88d0>, '_keep_checkpoint_every_n_hours': 10000, '_protocol': None, '_eval_distribute': None, '_num_worker_replicas': 1}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Not using Distribute Coordinator.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:30:01.025296 139814528628480 estimator_training.py:185] Not using Distribute Coordinator.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Running training and evaluation locally (non-distributed).\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:30:01.026371 139814528628480 training.py:610] Running training and evaluation locally (non-distributed).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps None or save_checkpoints_secs 600.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:30:01.027383 139814528628480 training.py:698] Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps None or save_checkpoints_secs 600.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Colocations handled automatically by placer.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0412 19:30:01.048937 139814528628480 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Colocations handled automatically by placer.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Calling model_fn.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:30:01.193914 139814528628480 estimator.py:1111] Calling model_fn.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column.py:205: EmbeddingColumn._get_dense_tensor (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed after 2018-11-30.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0412 19:30:01.196372 139814528628480 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column.py:205: EmbeddingColumn._get_dense_tensor (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed after 2018-11-30.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column_v2.py:3100: HashedCategoricalColumn._get_sparse_tensors (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed after 2018-11-30.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0412 19:30:01.197536 139814528628480 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column_v2.py:3100: HashedCategoricalColumn._get_sparse_tensors (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed after 2018-11-30.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column.py:2121: HashedCategoricalColumn._transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed after 2018-11-30.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0412 19:30:01.198594 139814528628480 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column.py:2121: HashedCategoricalColumn._transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed after 2018-11-30.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column_v2.py:3040: HashedCategoricalColumn._num_buckets (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed after 2018-11-30.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0412 19:30:01.207230 139814528628480 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column_v2.py:3040: HashedCategoricalColumn._num_buckets (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed after 2018-11-30.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column.py:206: EmbeddingColumn._variable_shape (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed after 2018-11-30.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0412 19:30:01.253056 139814528628480 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column.py:206: EmbeddingColumn._variable_shape (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed after 2018-11-30.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column.py:205: IndicatorColumn._get_dense_tensor (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed after 2018-11-30.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0412 19:30:01.260530 139814528628480 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column.py:205: IndicatorColumn._get_dense_tensor (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed after 2018-11-30.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column.py:2121: IndicatorColumn._transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed after 2018-11-30.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0412 19:30:01.261670 139814528628480 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column.py:2121: IndicatorColumn._transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed after 2018-11-30.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column_v2.py:4295: CrossedColumn._get_sparse_tensors (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed after 2018-11-30.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0412 19:30:01.262700 139814528628480 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column_v2.py:4295: CrossedColumn._get_sparse_tensors (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed after 2018-11-30.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column.py:2121: CrossedColumn._transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed after 2018-11-30.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0412 19:30:01.263766 139814528628480 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column.py:2121: CrossedColumn._transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed after 2018-11-30.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column_v2.py:4115: VocabularyFileCategoricalColumn._get_sparse_tensors (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed after 2018-11-30.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0412 19:30:01.264806 139814528628480 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column_v2.py:4115: VocabularyFileCategoricalColumn._get_sparse_tensors (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed after 2018-11-30.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column.py:2121: VocabularyFileCategoricalColumn._transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed after 2018-11-30.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0412 19:30:01.265817 139814528628480 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column.py:2121: VocabularyFileCategoricalColumn._transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed after 2018-11-30.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column_v2.py:4115: BucketizedColumn._get_sparse_tensors (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed after 2018-11-30.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0412 19:30:01.281163 139814528628480 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column_v2.py:4115: BucketizedColumn._get_sparse_tensors (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed after 2018-11-30.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column.py:2121: BucketizedColumn._transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed after 2018-11-30.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0412 19:30:01.282259 139814528628480 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column.py:2121: BucketizedColumn._transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed after 2018-11-30.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column.py:2121: NumericColumn._transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed after 2018-11-30.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0412 19:30:01.283296 139814528628480 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column.py:2121: NumericColumn._transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed after 2018-11-30.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column_v2.py:2703: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0412 19:30:01.286731 139814528628480 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column_v2.py:2703: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column_v2.py:2898: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0412 19:30:01.306609 139814528628480 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column_v2.py:2898: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column_v2.py:4266: IndicatorColumn._variable_shape (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed after 2018-11-30.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0412 19:30:01.314403 139814528628480 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column_v2.py:4266: IndicatorColumn._variable_shape (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed after 2018-11-30.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column_v2.py:4321: CrossedColumn._num_buckets (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed after 2018-11-30.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0412 19:30:01.315482 139814528628480 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column_v2.py:4321: CrossedColumn._num_buckets (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed after 2018-11-30.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column_v2.py:4321: VocabularyFileCategoricalColumn._num_buckets (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed after 2018-11-30.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0412 19:30:01.329684 139814528628480 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column_v2.py:4321: VocabularyFileCategoricalColumn._num_buckets (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed after 2018-11-30.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column.py:205: NumericColumn._get_dense_tensor (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed after 2018-11-30.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0412 19:30:01.392565 139814528628480 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column.py:205: NumericColumn._get_dense_tensor (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed after 2018-11-30.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column.py:206: NumericColumn._variable_shape (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed after 2018-11-30.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0412 19:30:01.395684 139814528628480 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column.py:206: NumericColumn._variable_shape (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed after 2018-11-30.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Saver not created because there are no variables in the graph to restore\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:30:01.479329 139814528628480 saver.py:1483] Saver not created because there are no variables in the graph to restore\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Saver not created because there are no variables in the graph to restore\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:30:01.523839 139814528628480 saver.py:1483] Saver not created because there are no variables in the graph to restore\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From <ipython-input-15-a06ef894ad16>:8: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use keras.layers.dense instead.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0412 19:30:01.597115 139814528628480 deprecation.py:323] From <ipython-input-15-a06ef894ad16>:8: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use keras.layers.dense instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
      "For more information, please see:\n",
      "  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
      "  * https://github.com/tensorflow/addons\n",
      "If you depend on functionality not listed there, please file an issue.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/metrics_impl.py:2295: to_double (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0412 19:30:03.523411 139814528628480 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/metrics_impl.py:2295: to_double (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/metrics_impl.py:3040: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Deprecated in favor of operator or tf.math.divide.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0412 19:30:03.532375 139814528628480 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/metrics_impl.py:3040: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Deprecated in favor of operator or tf.math.divide.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/array_grad.py:425: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0412 19:30:03.673475 139814528628480 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/array_grad.py:425: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Done calling model_fn.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:30:03.754856 139814528628480 estimator.py:1113] Done calling model_fn.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Create CheckpointSaverHook.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:30:03.758756 139814528628480 basic_session_run_hooks.py:527] Create CheckpointSaverHook.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Graph was finalized.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:30:04.048531 139814528628480 monitored_session.py:222] Graph was finalized.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Running local_init_op.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:30:04.453683 139814528628480 session_manager.py:491] Running local_init_op.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Done running local_init_op.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:30:05.282081 139814528628480 session_manager.py:493] Done running local_init_op.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Saving checkpoints for 0 into hybrid_recommendation_trained/model.ckpt.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:30:05.964669 139814528628480 basic_session_run_hooks.py:594] Saving checkpoints for 0 into hybrid_recommendation_trained/model.ckpt.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:loss = 9.658281, step = 1\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:30:07.654002 139814528628480 basic_session_run_hooks.py:249] loss = 9.658281, step = 1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:global_step/sec: 7.55423\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:30:20.891281 139814528628480 basic_session_run_hooks.py:680] global_step/sec: 7.55423\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:loss = 5.0486503, step = 101 (13.241 sec)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:30:20.894539 139814528628480 basic_session_run_hooks.py:247] loss = 5.0486503, step = 101 (13.241 sec)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:global_step/sec: 8.48253\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:30:32.680263 139814528628480 basic_session_run_hooks.py:680] global_step/sec: 8.48253\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:loss = 5.12706, step = 201 (11.789 sec)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:30:32.683902 139814528628480 basic_session_run_hooks.py:247] loss = 5.12706, step = 201 (11.789 sec)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:global_step/sec: 7.99466\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:30:45.188582 139814528628480 basic_session_run_hooks.py:680] global_step/sec: 7.99466\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:loss = 4.595773, step = 301 (12.508 sec)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:30:45.191941 139814528628480 basic_session_run_hooks.py:247] loss = 4.595773, step = 301 (12.508 sec)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:global_step/sec: 8.24176\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:30:57.321942 139814528628480 basic_session_run_hooks.py:680] global_step/sec: 8.24176\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:loss = 4.7829037, step = 401 (12.137 sec)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:30:57.329067 139814528628480 basic_session_run_hooks.py:247] loss = 4.7829037, step = 401 (12.137 sec)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:global_step/sec: 7.88129\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:31:10.010172 139814528628480 basic_session_run_hooks.py:680] global_step/sec: 7.88129\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:loss = 4.3145294, step = 501 (12.684 sec)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:31:10.013438 139814528628480 basic_session_run_hooks.py:247] loss = 4.3145294, step = 501 (12.684 sec)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:global_step/sec: 8.47436\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:31:21.810532 139814528628480 basic_session_run_hooks.py:680] global_step/sec: 8.47436\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:loss = 4.7389855, step = 601 (11.801 sec)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:31:21.814045 139814528628480 basic_session_run_hooks.py:247] loss = 4.7389855, step = 601 (11.801 sec)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:global_step/sec: 8.13935\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:31:34.096502 139814528628480 basic_session_run_hooks.py:680] global_step/sec: 8.13935\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:loss = 4.9688425, step = 701 (12.286 sec)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:31:34.100217 139814528628480 basic_session_run_hooks.py:247] loss = 4.9688425, step = 701 (12.286 sec)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:global_step/sec: 8.73241\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:31:45.548073 139814528628480 basic_session_run_hooks.py:680] global_step/sec: 8.73241\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:loss = 4.597124, step = 801 (11.451 sec)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:31:45.551150 139814528628480 basic_session_run_hooks.py:247] loss = 4.597124, step = 801 (11.451 sec)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:global_step/sec: 8.37975\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:31:57.481602 139814528628480 basic_session_run_hooks.py:680] global_step/sec: 8.37975\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:loss = 5.5182076, step = 901 (11.933 sec)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:31:57.484566 139814528628480 basic_session_run_hooks.py:247] loss = 5.5182076, step = 901 (11.933 sec)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Saving checkpoints for 1000 into hybrid_recommendation_trained/model.ckpt.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:32:10.842094 139814528628480 basic_session_run_hooks.py:594] Saving checkpoints for 1000 into hybrid_recommendation_trained/model.ckpt.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Calling model_fn.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:32:11.526900 139814528628480 estimator.py:1111] Calling model_fn.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Saver not created because there are no variables in the graph to restore\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:32:11.975584 139814528628480 saver.py:1483] Saver not created because there are no variables in the graph to restore\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Saver not created because there are no variables in the graph to restore\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:32:12.014522 139814528628480 saver.py:1483] Saver not created because there are no variables in the graph to restore\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Done calling model_fn.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:32:12.501098 139814528628480 estimator.py:1113] Done calling model_fn.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Starting evaluation at 2019-04-12T19:32:12Z\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:32:12.521400 139814528628480 evaluation.py:257] Starting evaluation at 2019-04-12T19:32:12Z\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Graph was finalized.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:32:12.616249 139814528628480 monitored_session.py:222] Graph was finalized.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use standard file APIs to check for files with this prefix.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0412 19:32:12.617941 139814528628480 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use standard file APIs to check for files with this prefix.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from hybrid_recommendation_trained/model.ckpt-1000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:32:12.620227 139814528628480 saver.py:1270] Restoring parameters from hybrid_recommendation_trained/model.ckpt-1000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Running local_init_op.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:32:12.810336 139814528628480 session_manager.py:491] Running local_init_op.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Done running local_init_op.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:32:13.577881 139814528628480 session_manager.py:493] Done running local_init_op.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Finished evaluation at 2019-04-12-19:32:24\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:32:24.890921 139814528628480 evaluation.py:277] Finished evaluation at 2019-04-12-19:32:24\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.02425876, global_step = 1000, loss = 5.461938, map_at_k = 0.054326190476190465, top_k_accuracy = 0.18133521\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:32:24.893011 139814528628480 estimator.py:1979] Saving dict for global step 1000: accuracy = 0.02425876, global_step = 1000, loss = 5.461938, map_at_k = 0.054326190476190465, top_k_accuracy = 0.18133521\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: hybrid_recommendation_trained/model.ckpt-1000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:32:25.018903 139814528628480 estimator.py:2039] Saving 'checkpoint_path' summary for global step 1000: hybrid_recommendation_trained/model.ckpt-1000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Calling model_fn.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:32:25.064716 139814528628480 estimator.py:1111] Calling model_fn.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Saver not created because there are no variables in the graph to restore\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:32:25.515646 139814528628480 saver.py:1483] Saver not created because there are no variables in the graph to restore\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Saver not created because there are no variables in the graph to restore\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:32:25.558695 139814528628480 saver.py:1483] Saver not created because there are no variables in the graph to restore\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Done calling model_fn.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:32:25.861397 139814528628480 estimator.py:1113] Done calling model_fn.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/saved_model/signature_def_utils_impl.py:205: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0412 19:32:25.863256 139814528628480 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/saved_model/signature_def_utils_impl.py:205: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Signatures INCLUDED in export for Eval: None\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:32:25.866004 139814528628480 export.py:587] Signatures INCLUDED in export for Eval: None\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict_export_outputs', 'serving_default']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:32:25.866916 139814528628480 export.py:587] Signatures INCLUDED in export for Predict: ['predict_export_outputs', 'serving_default']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:32:25.867748 139814528628480 export.py:587] Signatures INCLUDED in export for Regress: None\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:32:25.868529 139814528628480 export.py:587] Signatures INCLUDED in export for Classify: None\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Signatures INCLUDED in export for Train: None\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:32:25.870359 139814528628480 export.py:587] Signatures INCLUDED in export for Train: None\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from hybrid_recommendation_trained/model.ckpt-1000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:32:25.908200 139814528628480 saver.py:1270] Restoring parameters from hybrid_recommendation_trained/model.ckpt-1000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Assets added to graph.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:32:26.061472 139814528628480 builder_impl.py:654] Assets added to graph.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Assets written to: hybrid_recommendation_trained/export/exporter/temp-b'1555097545'/assets\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:32:26.128180 139814528628480 builder_impl.py:763] Assets written to: hybrid_recommendation_trained/export/exporter/temp-b'1555097545'/assets\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:SavedModel written to: hybrid_recommendation_trained/export/exporter/temp-b'1555097545'/saved_model.pb\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:32:26.484489 139814528628480 builder_impl.py:414] SavedModel written to: hybrid_recommendation_trained/export/exporter/temp-b'1555097545'/saved_model.pb\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Loss for final step: 4.4725347.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0412 19:32:26.913547 139814528628480 estimator.py:359] Loss for final step: 4.4725347.\n"
     ]
    }
   ],
   "source": [
    "# Call train and evaluate loop\n",
    "import shutil\n",
    "\n",
    "outdir = \"hybrid_recommendation_trained\"\n",
    "shutil.rmtree(path = outdir, ignore_errors = True) # start fresh each time\n",
    "\n",
    "arguments = {\n",
    "    \"bucket\": BUCKET,\n",
    "    \"train_data_paths\": \"gs://{}/hybrid_recommendation/preproc/features/train.csv*\".format(BUCKET),\n",
    "    \"eval_data_paths\": \"gs://{}/hybrid_recommendation/preproc/features/eval.csv*\".format(BUCKET),\n",
    "    \"output_dir\": outdir,\n",
    "    \"batch_size\": 128,\n",
    "    \"learning_rate\": 0.1,\n",
    "    \"hidden_units\": [256, 128, 64],\n",
    "    \"content_id_embedding_dimensions\": 10,\n",
    "    \"author_embedding_dimensions\": 10,\n",
    "    \"top_k\": 10,\n",
    "    \"train_steps\": 1000,\n",
    "    \"start_delay_secs\": 30,\n",
    "    \"throttle_secs\": 30\n",
    "}\n",
    "\n",
    "train_and_evaluate(arguments)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run on module locally\n",
    "\n",
    "Now let's place our code into a python module with model.py and task.py files so that we can train using Google Cloud's ML Engine! First, let's test our module locally."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile requirements.txt\n",
    "tensorflow_hub"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "echo \"bucket=${BUCKET}\"\n",
    "rm -rf hybrid_recommendation_trained\n",
    "export PYTHONPATH=${PYTHONPATH}:${PWD}/hybrid_recommendations_module\n",
    "python -m trainer.task \\\n",
    "    --bucket=${BUCKET} \\\n",
    "    --train_data_paths=gs://${BUCKET}/hybrid_recommendation/preproc/features/train.csv* \\\n",
    "    --eval_data_paths=gs://${BUCKET}/hybrid_recommendation/preproc/features/eval.csv* \\\n",
    "    --output_dir=${OUTDIR} \\\n",
    "    --batch_size=128 \\\n",
    "    --learning_rate=0.1 \\\n",
    "    --hidden_units=\"256 128 64\" \\\n",
    "    --content_id_embedding_dimensions=10 \\\n",
    "    --author_embedding_dimensions=10 \\\n",
    "    --top_k=10 \\\n",
    "    --train_steps=1000 \\\n",
    "    --start_delay_secs=30 \\\n",
    "    --throttle_secs=60"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Run on Google Cloud AI Platform\n",
    "If our module locally trained fine, let's now use of the power of AI Platform to scale it out on Google Cloud."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "OUTDIR=gs://${BUCKET}/hybrid_recommendation/small_trained_model\n",
    "JOBNAME=hybrid_recommendation_$(date -u +%y%m%d_%H%M%S)\n",
    "echo $OUTDIR $REGION $JOBNAME\n",
    "gcloud storage rm --recursive --continue-on-error $OUTDIR\n",    "gcloud ml-engine jobs submit training $JOBNAME \\\n",
    "    --region=$REGION \\\n",
    "    --module-name=trainer.task \\\n",
    "    --package-path=$(pwd)/hybrid_recommendations_module/trainer \\\n",
    "    --job-dir=$OUTDIR \\\n",
    "    --staging-bucket=gs://$BUCKET \\\n",
    "    --scale-tier=STANDARD_1 \\\n",
    "    --runtime-version=$TFVERSION \\\n",
    "    -- \\\n",
    "    --bucket=${BUCKET} \\\n",
    "    --train_data_paths=gs://${BUCKET}/hybrid_recommendation/preproc/features/train.csv* \\\n",
    "    --eval_data_paths=gs://${BUCKET}/hybrid_recommendation/preproc/features/eval.csv* \\\n",
    "    --output_dir=${OUTDIR} \\\n",
    "    --batch_size=128 \\\n",
    "    --learning_rate=0.1 \\\n",
    "    --hidden_units=\"256 128 64\" \\\n",
    "    --content_id_embedding_dimensions=10 \\\n",
    "    --author_embedding_dimensions=10 \\\n",
    "    --top_k=10 \\\n",
    "    --train_steps=1000 \\\n",
    "    --start_delay_secs=30 \\\n",
    "    --throttle_secs=30"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's add some hyperparameter tuning!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile hyperparam.yaml\n",
    "trainingInput:\n",
    "    hyperparameters:\n",
    "        goal: MAXIMIZE\n",
    "        maxTrials: 5\n",
    "        maxParallelTrials: 1\n",
    "        hyperparameterMetricTag: accuracy\n",
    "        params:\n",
    "            - parameterName: batch_size\n",
    "              type: INTEGER\n",
    "              minValue: 8\n",
    "              maxValue: 64\n",
    "              scaleType: UNIT_LINEAR_SCALE\n",
    "            - parameterName: learning_rate\n",
    "              type: DOUBLE\n",
    "              minValue: 0.01\n",
    "              maxValue: 0.1\n",
    "              scaleType: UNIT_LINEAR_SCALE\n",
    "            - parameterName: hidden_units\n",
    "              type: CATEGORICAL\n",
    "              categoricalValues: [\"1024 512 256\", \"1024 512 128\", \"1024 256 128\", \"512 256 128\", \"1024 512 64\", \"1024 256 64\", \"512 256 64\", \"1024 128 64\", \"512 128 64\", \"256 128 64\", \"1024 512 32\", \"1024 256 32\", \"512 256 32\", \"1024 128 32\", \"512 128 32\", \"256 128 32\", \"1024 64 32\", \"512 64 32\", \"256 64 32\", \"128 64 32\"]\n",
    "            - parameterName: content_id_embedding_dimensions\n",
    "              type: INTEGER\n",
    "              minValue: 5\n",
    "              maxValue: 250\n",
    "              scaleType: UNIT_LOG_SCALE\n",
    "            - parameterName: author_embedding_dimensions\n",
    "              type: INTEGER\n",
    "              minValue: 5\n",
    "              maxValue: 30\n",
    "              scaleType: UNIT_LINEAR_SCALE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "OUTDIR=gs://${BUCKET}/hybrid_recommendation/hypertuning\n",
    "JOBNAME=hybrid_recommendation_$(date -u +%y%m%d_%H%M%S)\n",
    "echo $OUTDIR $REGION $JOBNAME\n",
    "gcloud storage rm --recursive --continue-on-error $OUTDIR\n",    "gcloud ml-engine jobs submit training $JOBNAME \\\n",
    "    --region=$REGION \\\n",
    "    --module-name=trainer.task \\\n",
    "    --package-path=$(pwd)/hybrid_recommendations_module/trainer \\\n",
    "    --job-dir=$OUTDIR \\\n",
    "    --staging-bucket=gs://$BUCKET \\\n",
    "    --scale-tier=STANDARD_1 \\\n",
    "    --runtime-version=$TFVERSION \\\n",
    "    --config=hyperparam.yaml \\\n",
    "    -- \\\n",
    "    --bucket=${BUCKET} \\\n",
    "    --train_data_paths=gs://${BUCKET}/hybrid_recommendation/preproc/features/train.csv* \\\n",
    "    --eval_data_paths=gs://${BUCKET}/hybrid_recommendation/preproc/features/eval.csv* \\\n",
    "    --output_dir=${OUTDIR} \\\n",
    "    --batch_size=128 \\\n",
    "    --learning_rate=0.1 \\\n",
    "    --hidden_units=\"256 128 64\" \\\n",
    "    --content_id_embedding_dimensions=10 \\\n",
    "    --author_embedding_dimensions=10 \\\n",
    "    --top_k=10 \\\n",
    "    --train_steps=1000 \\\n",
    "    --start_delay_secs=30 \\\n",
    "    --throttle_secs=30"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we know the best hyperparameters, run a big training job!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "OUTDIR=gs://${BUCKET}/hybrid_recommendation/big_trained_model\n",
    "JOBNAME=hybrid_recommendation_$(date -u +%y%m%d_%H%M%S)\n",
    "echo $OUTDIR $REGION $JOBNAME\n",
    "gcloud storage rm --recursive --continue-on-error $OUTDIR\n",    "gcloud ml-engine jobs submit training $JOBNAME \\\n",
    "    --region=$REGION \\\n",
    "    --module-name=trainer.task \\\n",
    "    --package-path=$(pwd)/hybrid_recommendations_module/trainer \\\n",
    "    --job-dir=$OUTDIR \\\n",
    "    --staging-bucket=gs://$BUCKET \\\n",
    "    --scale-tier=STANDARD_1 \\\n",
    "    --runtime-version=$TFVERSION \\\n",
    "    -- \\\n",
    "    --bucket=${BUCKET} \\\n",
    "    --train_data_paths=gs://${BUCKET}/hybrid_recommendation/preproc/features/train.csv* \\\n",
    "    --eval_data_paths=gs://${BUCKET}/hybrid_recommendation/preproc/features/eval.csv* \\\n",
    "    --output_dir=${OUTDIR} \\\n",
    "    --batch_size=128 \\\n",
    "    --learning_rate=0.1 \\\n",
    "    --hidden_units=\"256 128 64\" \\\n",
    "    --content_id_embedding_dimensions=10 \\\n",
    "    --author_embedding_dimensions=10 \\\n",
    "    --top_k=10 \\\n",
    "    --train_steps=10000 \\\n",
    "    --start_delay_secs=30 \\\n",
    "    --throttle_secs=30"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
